1. Field of the Invention
Embodiments of the present invention relate to a digital photo album, and more particularly, to a method, medium, and apparatus for person-based photo clustering, and a person-based digital photo albuming method, medium, and apparatus.
2. Description of the Related Art
Recently, demands for digital cameras have been increasing more and more. In particular, with the recent development of memory technologies, highly integrated ultra-small-sized memory devices are now widely used. Further, with the development of digital image compression technologies that do not compromise picture quality, users can now store hundreds to thousands of photos in one memory device. As a result, many users now need apparatuses and tools to more effectively manage many photos. Accordingly, demands by users for an efficient digital photo album are increasing.
Generally, a digital photo album can be used to transfer photos taken by a user from a digital camera or a memory card to a local storage apparatus of the user and to manage the photos conveniently in a computer. Users may cluster and index many photos in a time series, by person, or even by specific category, using the photo album. A correspondingly constructed photo album enables the users to easily browse photo data or share photos with other users later.
In “Requirement For Photoware” (ACM CSCW, 2002), David Frohlich investigated the photo album preferred capabilities desired by users through a survey of many users. Most interviewees agreed with the necessity of a digital photo album, but felt that the time and effort necessary to group or label many photos, one by one, was inconvenient, and expressed difficulties in sharing photos with others.
So far, there have been quite a bit of research suggesting differing methods of clustering photo data. Basically, those methods include clustering photos according to the photographed time, and methods of clustering photos based on contents-based feature information. As further developments, research on methods of clustering photos based on events of the photos, and methods of clustering photos based on the actual people in the photos, have been carried out.
Among these methods, one of the methods that users most frequently use is clustering of photo data by person, i.e., clustering based on the people in the photo. Automatic clustering of a large volume of photo data in relation to persons taken in the photos allows users to easily share photo data with others or easily browse photo data later.
The most important technology in the process of clustering photo data by person is by use of face recognition technology. Face recognition can be explained as a process of building a facial database containing facial data of different faces of different people, and then, comparing an input facial image with facial image data stored in the already built database to determine who the input facial image belongs to.
That is, in this process, a facial image can be detected within photo data and, by comparing the facial image with facial image data stored in the database, the input facial image can be recognized. For this, a process almost identical to that of performing pattern recognition in an image processing method is performed, with such a method including image acquisition, pre-processing, face extraction, and face recognition.
A lot of research has been performed on methods for detecting and/or recognizing a face in an image. The face recognition field has primarily been developed centered on security systems. For example, research has mainly been performed on face extraction and recognition for a automated intelligent monitoring system, an entrance control system, and a criminal suspects retrieval system. In these applications, the research has focused on a facial recognition method that is robust against external lighting, facial expression and pose of a face in which the face of a person can be more accurately extracted.
Unlike this, corresponding research on recognizing a face in photo data is in its fledgling stage. The process of extracting or recognizing a face in photo data is much more difficult than that of extracting or recognizing a face in an image obtained by a security system. Since the security system obtains image data by using a camera fixedly installed in a fixed space, a facial image extracted from an image has a relatively typical background. On the other hand, since photo data includes images taken by a user at different places, backgrounds in corresponding photos are diverse, frequently changing. In addition, as a camera may be used in different manners (e.g., though use of a zoom function or a flash), or the direction of a camera when a photo is taken may change, a variety of backgrounds can be shown in an identical space, and changes of external lighting or illumination are much greater.
Due to the reasons described above, in a method of clustering photo data by person, if only facial images are used, as in the conventional security system, the method results in very low level performance. In order to soundly cluster photo data by person, a method and system using a variety of additional information items that can be obtained from photos, in addition to facial image information extracted from photos, thus, are required.
As leading research to solve this problem, the following is representative of such research. In “Face Recognition-Matching System Effective to Images Obtained in Different Imaging Conditions” (U.S. Pat. No. 6,345,109 B1), a system capable of recognizing a facial image having serious noise, with a front facial image having relatively smaller interference from lighting, is suggested. However, this system has a problem that a user must prepare facial images having less noise in advance.
In “Automatic Cataloging of People in Digital Photographs” (U.S. Pat. No. 6,606,398 B2), identification parameters related to individual facial images stored in a database are defined and these are then used for face-based photo clustering. As identification parameters, there are names, relations, sexes, ages, and so on. A newly input image is found in the database by using an identification parameter input by the user, and then clustered. However, since the user must directly input the identification parameter, a great amount of time is required for inputting the identification parameters, and therefore this method cannot be practically used.
In “Automatic Face-based Image Grouping for Albuming” (IEEE, 2003), suggested is a method by which the age and sex of each face are automatically detected, and by using these, a group of photos are face-based clustered. This method has an advantage that age and sex information of a person, as well as facial information of a photo, are used, but if a variety of additional information items of a photo are used together, more effective photo grouping can be performed.
That is, an appropriate combination of a variety of additional information items that can be obtained from photo data and consideration of elements that can occur in a photo image, such as external lighting change, pose change, facial expression change, and time change, are needed. If features of photos, for example, the feature of photo data being taken in shorter time intervals, similarities between backgrounds and similarities between people may be considered, along with similarities between worn clothes, the performance of person-based photo clustering may be greatly improved over that of photo clustering using only facial information, as will be discussed in greater detail below.